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1. Introduction 



By fall 2005, the DC Opportunity Scholarship Program (OSP), the first federally funded 
voucher program in the United States, was in its second year serving low-income students in the 
nation’s capital. More than 5,800 students have applied to the Program over the 2 years, 
and about 2,300 of them — eligible public school students who participated in a lottery to 
determine scholarship award — are the subject of a rigorous impact evaluation mandated by the 
Program statute. While the most important questions for the evaluation are about the 
Program’s effectiveness in improving student outcomes, data are still being collected for that 
analysis and will be presented in a 2007 report. This document from the study team provides a 
brief update to the first report to Congress^ by describing the schools and students who applied to 
and became participants in the Program for the 2005-06 school year. The analysis indicates that 
by fall 2005, the Program was operating at capacity, with more than 1,700 students using 
scholarships at 60 of 68 participating private schools. 

1.1 The Program 

The District of Columbia School Choice Incentive Act of 2003 was passed by Congress in 
January 2004. The Act provided funds for District of Columbia Public Schools’ 
(DCPS) improvement activities and charter school facility acquisitions. Most notably, 
the statute established what is now called the DC Opportunity Scholarship Program — the first 
Federal government initiative to provide K-12 education scholarships, or vouchers, to families 
to send their children to private schools of choice. 

The DC Opportunity Scholarship Program has the following programmatic elements: 

• To be eligible, students entering grades K-12 must reside in the District and have a 
family income at or below 185 percent of the Federal poverty line. 

• Participating students receive scholarships of up to $7,500 to cover the costs of 
tuition, school fees, and transportation to a participating private school of choice. 

• Scholarships are renewable for up to 5 years (as funds are appropriated), so long as 
students remain eligible for the Program and remain in good academic standing at the 
private schools they are attending. 

• If there are more eligible applicants than available scholarships or open slots in 
private schools, applicants are to be awarded scholarships and admission to private 
schools by random selection, for example, by lottery. 

• In making these scholarship awards, priority is given to students attending public 
schools designated as in need of improvement (SINI) under the No Child Left 
Behind (NCLB) Act and to families that lack the resources to take advantage of 
school choice options. 

• Private schools participating in the Program must be located in the District and 
must agree to Program requirements regarding nondiscrimination in admissions, 
fiscal accountability, and cooperation with the evaluation. 



' To access the first evaluation report, see http://www.ed.gov/ies/ncee . 
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Implementation of the OSP, as analyzed in this report, took place over an 18-month period from 
April 2004 to September 2005. In late March 2004, the Washington Scholarship Fund (WSF), a 
501(c)3 organization in the District of Columbia, was selected by the U.S. Department 
of Education (ED) to implement the OSP, under the supervision of the Office of Innovation and 
Improvement in ED and the Office of the Mayor of the District of Columbia. Since then, the 
WSE has worked with its implementation partners^ to finalize the Program design, establish 
protocols, recruit applicants and schools, and place scholarship winners in participating private 
schools. 

The funds appropriated for the OSP are sufficient to support approximately 1,700-1,800 students, 
depending on the cost of the participating private schools that they attend. Between students 
recruited during the first year who are continuing to use their scholarships and those awarded and 
using scholarships from the second year of recruitment, the Program is now operating at full 
capacity. 

1.2 The Mandated Evaluation 

The Act requires that this 5-year scholarship pilot Program be rigorously evaluated by an 
independent research team, using the “strongest possible research design for determining the 
effectiveness” of the Program and addressing a specific set of student comparisons and topics 
(Section 309): 

• Impact Analysis. Central to the evaluation is an impact analysis that compares 
outcomes of eligible applicants (students and their parents) from public schools 
randomly assigned to receive or not receive a scholarship through a lottery. Such 
random assignment experimental designs are widely viewed as the best 
methods for identifying the independent effect of programs on subsequent 
outcomes.^ Thus, the impact analysis will be the source of the reliable, causal 
evidence on Program effectiveness called for in the legislation (see appendix A for a 
more comprehensive description of the evaluation and its technical approach). 

• Performance Reporting. The Act also specifies a comparison of students 
participating in the scholarship Program with students in the same grades in the 
DCPS system, as a way of tracking general student progress and Program 
performance."^ Such a comparison would draw upon what we call the “OSP recipient 
sample,” which comprises all students offered a scholarship, including students who 
were already attending private schools at the point of application and public school 



^ The WSF has joined with Capital Partners for Education, DC Parents for School Choice, and Fight for Children — all District- 
based nonprofit organizations, to assist in client recruitment and implementation activities. 

^ For examples, see the What Works Clearinghouse, WWC Study Review Standards, 7 (http://www.whatworks.ed.gov/ 
reviewprocess/studv_standards_final.pdf); Thomas D. Cook and Monique R. Payne, “Objecting to the Objections to Using 
Random Assignment in Educational Research,” in Evidence Matters: Randomized Trials in Education Research, eds. 
Frederick Mosteller and Robert Baruch (Washington, DC: Brookings, 2002). 

DCPS students who did not apply to the scholarship Program are likely to be quite different from those who applied and are 
participating in the OSP — in ways we can observe and ways we cannot. Comparing outcomes between participants and 
nonapplicants is, therefore, not a reliable measure of Program effects. 
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applicants who were automatically awarded scholarships.^’^ However, since the 
passage of the legislation and the first year of OSP implementation, DCPS has been 
in transition to a new academic assessment that differs from its earlier test, which the 

■7 

evaluation is required to use for its main outcomes measurement. The divergence 
between the new DCPS assessment and the evaluation assessment means that 
comparing the academic performance of all scholarship recipients and other DCPS 
students is no longer possible, although this analysis was performed for students 
who participated in the Program’s first year when the same assessment was used 
(see the first report to Congress). 

• Response of Schools. Through descriptive analyses, the evaluation will assess 
how DC public and private schools are changing during the implementation of 
the OSP, in part by examining the extent to which the schools are experiencing 
significant losses or gains in student enrollment during this period. 

1.3 Summary of Key Findings on Program Participation 

Over the first 18 months of implementation that ended September 2005, applications were 
accepted in essentially two waves: in spring 2004 for fall 2004 enrollment, which we call “cohort 
1,” and through spring 2005 for fall 2005 enrollment, which we call “cohort 2.” A total of 5,818 
students applied, and 4,047 of them were deemed eligible for the OSP. By fall 2005, 2,454 
students had been awarded scholarships, most by lottery because they were in grades for which 
there were more applicants than slots in participating private schools (table 1-1). 



Table 1-1. OSP applicants by program status, cohorts 1 and 2 





Cohort 1 


Cohort 2 


Total 


Applicants 


2,692 


3,126 


5,818 


Eligible applicants 


1,848 


2,199 


4,047 


In impact sample 


492 


1,816 


2,308 


Scholarship recipients 


1,366 


1,088 


2,454 


Scholarship users in initial year of receipt 


1,027 


797 


1,824 


Scholarship users in fall 2005 


919 


797 


1,716 



NOTES: Applicants entering grades 6-12 in cohort 2 who did not participate in 

baseline testing were not included in the eligible applicant figure. The initial 
year of receipt is fall 2004 for cohort 1 and fall 2005 for cohort 2. 

SOURCES: The DC Opportunity Scholarship Program applications and the Program 

operator’s files. 



^ Automatic scholarship awards were given only in the first year of Program implementation to all students applying from 
public schools designated “in need of improvement” under the 2002 reauthorization of the Elementary and Secondary 
Education Act and to all public school applicants entering grades K-5 that were not oversubscribed and therefore not subject 
to award by lottery. 

^ The “recipient sample” is different from the “impact sample,” which is limited to public school applicants who were subject to 
scholarship award by lottery and thus were randomly assigned to the “treatment” (scholarship) or “control” (nonscholarship) 
group. 

’ See Section 309(a)(3)(B) for the provision that stipulates that the evaluation use the same assessment as DCPS administered 
to public school students in the first year the OSP was operating. This requirement was intended to ensure that the impact 
analysis could be based on a consistent measure of student achievement and not subject to changes in the key outcome 
measure throughout the evaluation period. 
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In addition: 



. Private Schools. Ten new schools agreed to participate in the Program in 
the second year of operation, raising the total from 58 who signed on in the first 
year to 68 in fall 2005. The new schools tend to have smaller class sizes, higher 
regular tuitions, and smaller minority populations and are less likely to be Catholic- 
affiliated than the population of schools that have participated in the OSP from the 
start (table 2-1). Of the 68 schools participating in year 2, 60 were serving 
scholarship students in fall 2005. The remaining eight schools had no OSP students 
in fall 2005. Although systematic data were not collected as to the reason(s) each 
school did not serve OSP students, program implementation staff reported that the 
most common reasons include that schools either: (1) determined that none of the 
current scholarship recipients met their entrance criteria, (2) had no scholarship 
recipients choose their school during the placement phase, or (3) filled their vacant 
slots before OSP recipients could be placed, [section 2] 

. Background of Applicants. About 44 percent of total public school applicants to the 
Program came from a public school that was designated as SINI between 2003 and 
2005. Eleven percent came from the worst performing public schools, those in 
which the percentage of students who reached the “proficient” benchmark 
on the DCPS assessment placed them in the bottom quartile of all DCPS schools. 
On the other hand, almost one-quarter of all applicants are from the highest 
performing (top quartile of) DCPS schools based on proficiency rates, [section 3] 

• Impact Sample. A large subset of applicants during the first 2 years — 2,308 — were 
public school students who applied to be in grades for which there were more 
applicants than there were slots in participating private schools. Thus, a lottery 
determined whether they received a scholarship offer. These students were 
randomly assigned such that 1,387 received scholarships (treatment group) and 921 
did not receive scholarships (the control group). They make up the “impact sample.” 
Preliminary power analyses indicate that the impact sample is sufficient in number 
for the evaluation to be able to statistically detect meaningful and policy relevant 
differences in subsequent outcomes between the two groups, [section 3] 

. Characteristics of Treatment vs. Control Groups. The treatment and control 
groups are statistically similar on all but 2 out of 15 important baseline characteristics 
that could be measured (table 3-5). In year 2, for students entering grades K-5, the 
average family income and years of mother’s education are somewhat higher 
for the control group than for the treatment group. These differences are 
small and likely due to random chance, particularly since multiple small-scale 
lotteries were run for each grade band during year 2 in order to accommodate early 
and late applicants.^ In estimating Program impacts, the evaluation will use baseline 
measures of student background factors to control for these pre-Program differences, 
[section 3] 

. Scholarship Use Rates. Overall, 1,824 (74 percent) students who were awarded 
scholarships used them the initial year to attend a private school, although the rate 
was slightly lower for the impact sample of randomized public school applicants 



The smaller the number of students randomized at any time, the higher the odds of obtaining some statistical differences 
between the treatment and control groups. 
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(71 percent or 982 students). Lower use among the impact sample reflects two 
factors: (1) the group excludes students already attending private schools at the time 
of application, whose use rates are substantially higher than those for public school 
recipients and (2) the sample includes a higher proportion of older students, a group 
that was more constrained in their choice of schools under the Program and who 
experienced substantially lower use rates. Among cohort 1 students, there is an 8 
percentage point decline in use between the first and second years of scholarship 
award for both the overall and impact sample groups. Taking these use patterns 
into account, in September 2005, a total of 1,716 students were enrolled in private 
schools of their parent’s choosing by way of Opportunity Scholarships, [section 3] 



1.4 Organization of This Report 

The remainder of this report provides some additional details about participation as of the second 
year of Program implementation. Section 2 focuses on the DC private schools that offered to 
accept scholarship students during the first and second years of implementation. Section 
3 updates the number of students that applied to and were awarded scholarships as part of 
the Program, including the subset of applicants who are the focus of the upcoming impact 
analysis. 

For this report, which is descriptive, as well as for the later impact analysis reports, we will use 
several tests for calculating statistical significance, or the level of confidence that evaluators have 
that a difference between groups did not occur merely by chance. For most of the comparisons 
that we make, we use the “Student’s t test.” The t test is commonly used when the factor being 
considered, such as test scores, tends to be distributed continuously on a normal, bell-shaped 
curve. Unlike some significance tests, the t test incorporates information about the distribution of 
values in both comparison groups, and not just the overall population, and thus is a more precise 
measure of statistical significance than the Z test, for example.^ When the characteristic in 
question is not normally distributed — such as gender, which is an either/or and not a more-or- 
less — we use the “chi-squared” test of statistical significance. All group differences that are 
mentioned in this report are statistically significant at least beyond the traditional 95% 
confidence level using a two-tailed statistical test. 



* See Russell A. Langley, Practical Statistics Simply Explained (New York: Dover, 1970), 160-165. 
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2. Participating Schools 



The DC Opportunity Scholarship Program seeks to enable low-income parents in the District to 
send their children to private schools as an alternative to the public school or schools otherwise 
available to them. As such, one important characteristic of the Program is the composition of the 
set of DC private schools that chose to accept OSP students. 

Over half of District private schools have agreed to participate in the OSP. 

• 58 (53 percent) of the 109 private elementary and secondary schools in DC in 2004 
agreed to participate in the Program in the first year of implementation. 

• 68 (65 percent) of the 104 District private schools in 2005 — including all the schools 
that participated in the first year — chose to participate in the OSP during the second 
year of implementation. 

• Of the 68 participating schools in fall 2005, 60 (88 percent) had OSP students 
enrolled at that time. 

The religious status and affiliation of the participating schools varies (figure 2-1). Of the 63 
participating schools for which a religious status could be determined, 28 (44 percent) are 
Catholic, 14 (22 percent) are formally affiliated with a religion besides Catholicism, and 21 (33 
percent) are independent private schools. Perhaps because 28 of 30 DC private schools that 
identify themselves as Catholic all joined the Program the first year, none of the schools that 
joined in year 2 are; 4 of the 10 new private schools are non-Catholic religious schools, whereas 
the other 6 are independent. We were unable to determine the religious status of five schools, 
which all joined the Program in year l.'° 

The 10 schools that joined the Program in year 2 are different in several other respects from the 
58 schools that have participated from the start. On average, the new schools are more likely to 
charge tuition above the scholarship cap of $7,500, more likely to serve one or more high school 
grades, have a smaller percentage of racial minorities among their student populations, and are 
larger than the original group of participating schools (table 2-1). The average teacher/student 
ratios of the two groups of participating schools are statistically similar. 

The 36 District private schools that are not currently participating in the Program differ from the 
total set of participating schools in some respects. Nonparticipating schools are more likely to 
charge average tuitions above the scholarship cap, have smaller enrollments, serve a smaller 
minority population, and have lower student/teacher ratios than the average among the 
participating schools (table 2-1). The group of nonparticipating private schools includes 
several highly specialized schools, such as a ballet school, as well as schools that 
exclusively serve students with significant disabilities. 



We were unable to identify conclusively the religious status of five participating private schools because the field for 
“Religious Affiliation” in the WSF School Directory database was blank for those schools, and visits to their websites did not 
provide sufficient information to determine their religious status. 
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Figure 2-1. Religious affiliation of participating schools by year 
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NOTE: Schools are defined as “participating” if they signed an agreement form to accept 

scholarship students. No schools dropped out of the Program between 2004-05 and 
2005-06. Five Schools Did Not Identify Their Religious Affiliation In Either Year. 

SOURCES: “School Directory, D.C. K-12 Scholarship Program, 2004-05 School Year,” 
Washington Scholarship Fund, lune 2004. “School Directory, D.C. Opportunity 
Scholarship Program, 2005-06 School Year,” Washington Scholarship Fund, 
August 2005. 
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Table 2-1. Features of DC private schools by OSP participation status, years 1 and 2 



Item 


Participated 
year 1 


New participants 
year 2 


Total 

participants 
year 2 


Nonparticipants 
year 2 


Percent with average tuition^ 
above $7,500 


31.0'’ 


88.9** 


38.8 


73.7** 


Average size (student 
enrollment) 


204.0 


418.4** 


236.0 


137.6* 


Percent serving high schooP 


17.2 


50.0* 


22.1 


32.4 


Average percent minority 


81.2 


35.3** 


76.2 


56.6* 


Average student/teacher 
Ratio 


10.9 


8.5 


10.6 


7.8* 


Total N 


58 


10 


68 


36 



SOURCES: Data on participating private schools drawn from “School Directory, D.C. K-12 Scholarship Program, 2004-05 
School Year,” and “School Directory 2005-06, D.C. Opportunity Scholarship Program,” Washington Scholarship 
Fund, June 2004 and April 2005, respectively. Data on both participating and nonparticipating private schools were 
also obtained from school websites. 

For schools that charge a range of tuitions, the midpoint of the range was selected. Tuition rates were unavailable for 8 of the 
participating private schools and 26 of the nonparticipating private schools. 

’’ Three schools charged no tuition either because of foundation support or because the school serves groups such as DC-placed 
special education students funded by the government. 

Schools were classified as serving high schools if they enrolled students in any grade 9-12. 

* The difference between groups is statistically significant at the 95 percent confidence level. 

** The difference between groups is statistically significant at the 99 percent confidence level. 



Of the 68 DC private schools that agreed to participate, 60 had OSP students enrolled in fall 
2005. Eight schools were not serving OSP students at that time. Although systematic data 
were not collected as to the reason(s) each school did not serve OSP students, program 
implementation staff reported that the most common reasons include that schools: (1) determined 
that none of the current scholarship recipients met their entrance criteria, (2) had no scholarship 
recipients choose their school during the placement phase, or (3) filled their vacant slots before 
OSP recipients could be placed. 



’ * According to the Program statute, participating private schools are able to maintain any admittance criteria that they apply to 
non-OSP students in determining which OSP students are admissible. These criteria most typically include the completion 
of an interview with school staff, a placement test to determine their appropriate grade level, and an agreement to conform to 
the schools’ codes of conduct. However, 13 percent of participating schools require standardized testing results to be a part of 
the admissions packet, for both OSP and non-OSP students. 
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3. Families/Students 



How many and what kinds of people seek to avail themselves of an educational intervention 
signals both the levels and composition of client demand for such a program. This section 
describes the applicants to the Program, the numbers and types of applicants who were 
awarded scholarships, the size and composition of the randomized impact sample that will 
inform the evaluation, and the rates of scholarship usage and persistence in the Program. 

3.1 Applicants 

The Program received new applications on behalf of 3,126 students during the second year of 
implementation (table 3-1). In total, then, 5,818 distinct individuals applied for 
Opportunity Scholarships during the first two application periods. Over 2,000 new applicants 
were deemed eligible for the Program in cohort 2, for a total of 4,047 eligible applicants to date. 
This total represents approximately 10 percent of eligible low-income students in the District of 
Columbia, according to 2000 census figures. 

Table 3-1. Number and percentage of applicants, by application status, 
cohorts 1 and 2 



Measure 


Cohort 1 


Cohort 2 


Total 


Low-income students in DC 


40,507 


40,507 


40,507 


Applicants 


2,692 


3,126 


5,818 


Eligible applicants 


1,848 


2,199 


4,047 


Eligible applicants as percent of low-income students in DC 


5 


5 


10 



NOTE: Applicants entering grades 6-12 in cohort 2 who did not participate in baseline testing were not included 

in the eligible applicant figure. 

SOURCES: Figure for low-income students is based on data from the U.S. Census, population of the District of Columbia 
ages 5 to 17 under 185 percent of the Federal poverty line in 2000. The exact number for 2004 and 2005 is 
likely to differ somewhat from this 2000 figure. Numbers of applicants and eligible applicants are from the 
DC Opportunity Scholarship Program applications. 



Congress signaled its intention to ensure that the OSP was targeted to students most in need of 
educational options. In particular. Section 306 of the statute identifies students attending public 
schools officially designated as in need of improvement under NCLB as a group that 
should receive priority for the Program. This priority was implemented in both how 
students were recruited to apply to the Program and how scholarships were awarded through 
lotteries (see also section 3.2). Because the lotteries must be conducted in the spring, before the 



Under NCLB, schools are designated as SINI if they have been conducting accountability testing consistent with the 
requirements of NCLB for at least 2 years and contained a student population or relevant subpopulation that failed to 
demonstrate Adequate Yearly Progress in both years. 



9 





District reports its SINI designations each August, the lottery priority group categories are 
always based on SINI designations that are a year behind. For the purposes of examining 
applicant characteristics, however, it is more accurate to consider the designation for the school 
year in which a student applies to the OSP: 2004 SINI designations for cohort I (spring 2004 
applicants) and the 2005 designations for cohort 2 (spring 2005 applicants). In evaluating the 
extent to which the Program is drawing students with educational needs, we also use 
another indicator of school performance — the proportion of each applicant’s school that 
met the “proficient” benchmark on the DCPS assessment in 2005. 

Based on these different measures of educational need (table 3-2): 

• About 40 percent of applicants were from schools designated SINI for the year the 
students would be leaving those schools to participate in the OSP. For cohort 1, 37 
percent were SINI, and for cohort 2 just under 43 percent were SINI. In total, 44 
percent of OSP applicants were from schools designated as SINI between 2003 and 
2005, a period when the number of SINI schools jumped from 15 to 101. 

• About 1 1 percent of the applicants came from schools whose school-level 
proficiency rate in 2005 places them in the bottom quartile of all DCPS schools; 
nearly 30 percent were from the second quartile schools; and almost 36 percent were 
from third quartile schools. Just under one-quarter of all OSP applicants came from 
the highest performing public schools in the District, based on the proportion of 
the schools’ students who met the proficiency benchmark. 

3.2 Scholarships Awarded 

The Program statute set two conditions for any lotteries conducted under the OSP. First, 
scholarships would be awarded by lottery when the Program is “oversubscribed” — that is, the 
number of eligible applicants exceeds the number of available slots in participating private 
schools. However, because the extent of oversubscription varied significantly by grade, in 
practice the determination of whether to hold a lottery was considered within grade bands: those 
applying for grades K-5, those applying for grades 6-8, and those applying for grades 9-12. 
Second, the statute specified that certain groups be given priority in any such lotteries, which led 
to the following classifications for the lottery: 

• Applicants attending a public school in need of improvement under NCLB (highest 
priority); 

• Non-SINI public applicants (middle priority); and 

• Applicants already attending private schools (lowest priority). 



For example, SINI students in the cohort I (spring 2004) lottery had to be categorized based on SINI designations made in 
August 2003, using performance data from the 2002-03 school year. But 2 months after the lottery, more schools were 
designated SINI for 2004, and it is this later number that is a more accurate representation of which schools were low 
performing when cohort 1 was applying to the OSP. 
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Table 3-2. School-level academic performance designations and rates, public school 
attending at time of application to the OSP, cohorts 1 and 2 



Percent of OSP applicants 


Total 


Cohort 1 


Cohort 2 


Attending SINI schools 


SINI in fall 2003 (N=15) 


4.9 


5.9 


4.2 


SINI in fall 2004 (N=90) 


36.9 


37.1 


36.7 


SINI in fall 2005 (A=101) 


43.7 


44.8 


42.8 


Attending schools in which the percent of 
students proficient in math and reading 
placed the school in the: 


Lowest quartile 


11.2 


11.4 


11.1 


Second quartile 


29.5 


30.3 


28.9 


Third quartile 


35.6 


35.2 


35.9 


Highest quartile 


23.7 


23.3 


24.0 


Percent missing 


21 


19 


22 


Sample size 


3,159 


1,343 


1,816 



NOTES: The figures in bold in the upper panel are those most relevant for each cohort. Proficiency rates were 

calculated based on a composite measure of reading and mathematics rates. The rates were calculated for 
each DCPS school, and the schools were sorted by rate. The proficiency rate for the school that was the “25* 
percentile” school in the distribution became the cut-point for the lowest quartile. The rates for the schools 
at the 50* and 75* percentiles became the cut-points for the second, third, and highest quartiles. As context 
for interpreting this distribution, schools that fell in the two lowest quartiles had fewer than 44 percent of their 
students achieving proficiency on the DCPS assessment. 

SOURCES: The DC Opportunity Scholarship Program applications and the District of Columbia Public Schools website. 



These conditions led to somewhat different scholarship award opportunities in years 1 and 2 of 
Program implementation (figure 3-1). In cohort 1, for example, there were more slots in 
participating schools than there were applicants for grades K-5; therefore, all eligible K-5 
applicants automatically received scholarships, and no lotteries were conducted at that level. By 
contrast, in cohort 2 there were 1,178 K-5 applicants vying for scholarships to fill 643 slots, 
requiring a lottery to determine which students received an award. In both years, there was a high 
degree of oversubscription at the high school level and moderate oversubscription at the middle 
school level. In contrast to the first year, in year 2 there was sufficient demand from public 
school applicants that lotteries were conducted only for them; applicants who were 
already attending a private school (the lowest priority group) were not entered into a lottery. 
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Figure 3-1. Eligible public school applicants and available private school slots, by grade- 
level band, cohorts 1 and 2 




SOURCES: The DC Opportunity Scholarship Program applications and the Program operator’s files. 



The design of the lotteries for cohort 2 applicants was similar to that for the first year, with the 
exception of how the priority treatment of SINI applicants was implemented. For cohort 1, the 79 
eligible applicants from previously designated SINI schools were all automatically awarded 
scholarships, consistent with their status as the highest priority client group. For cohort 2, the 655 
eligible applicants from previously designated SINI schools were all subject to lotteries, with 
scholarship award probabilities approximately one-third higher than non-SINI applicants within 
their respective grade bands (table 3-3).^“^ Randomly assigning cohort 2 SINI students, but with a 
higher probability of award than non-SINI students, accomplished the dual objectives of 
including a significant number of SINI students in the experimental evaluation of Program 
impact while also treating them as the highest priority for access to the Program. 

In general, the probability of receiving a scholarship through a lottery was based on the 
ratio of slots to applicants in each grade band. Given the likelihood that some students would 
choose not to use the scholarships that were awarded to them, based on the first-year 
experience, award probabilities were then adjusted to “over-award” scholarships by 
approximately 20 percent. As a result of the spring 2005 lotteries, 1,088 scholarships were 
awarded to cohort 2 eligible applicants — 429 to applicants from SINI public schools and 
659 to the larger group of applicants from non-SINI public schools (table 3-3). 



The cohort 2 SINI award probability advantage actually was 32.7 percent in K-5, 35.6 percent in 6-8, and 33.5 percent in 9- 
12. In the end, it did not equal exactly 33.3 percent in any of the grade bands because a handful of students deemed eligible on 
appeal were randomly assigned through small lotteries that are inherently less precise than are large lotteries. 
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Table 3-3. Probability of receiving a scholarship, by applicant type and grade-level 
band, cohort 2 







Grade-level band 






Type of applicant 


K-5 


6-8 


9-12 


Total 


SINI public 


Eligibles 


277 


184 


194 


655 


Probability 


78.3% 


75.0% 


38.1% 


65.5% 


Scholarships 


217 


138 


74 


429 


Nonrecipients 


60 


46 


120 


226 


Non-SINI public 


Eligibles 


901 


197 


63 


1,161 


Probability 


59.0% 


55.3% 


28.6% 


56.8% 


Scholarships 


532 


109 


18 


659 


Nonrecipients 


369 


88 


45 


502 


All public 


Eligibles 


1,178 


381 


257 


1,816 


Probability 


63.6% 


64.8% 


35.8% 


59.9% 


Scholarships 


749 


247 


92 


1,088 


Nonrecipients 


429 


134 


165 


728 


Private 


Eligibles 


151 


122 


110 


383 


Probability 


0% 


0% 


0% 


0% 


Scholarships 


0 


0 


0 


0 


Nonrecipients 


151 


122 


110 


383 


Total (public and private) 


Eligibles 


1,329 


503 


367 


2,199 


Probability 


56.4% 


49.1% 


25.1% 


49.5% 


Scholarships 


749 


247 


92 


1,088 


Nonrecipients 


580 


256 


275 


1,111 



NOTES: Eligible applicants already attending private school were given a zero probability of receiving a 

scholarship because the Program was rapidly filling with higher priority public school applicants. 

Applicants in cohort 2 entering grades 6-12 were required to participate in baseline testing prior to entry 
in the lottery. As a result, applicants who were deemed eligible based on income and residency but did 
not participate in the baseline testing are not included in the table. 

SOURCES: The DC Opportunity Scholarship Program applications and the Program operator’s files. 

In total, then, after 18 months of Program implementation, the OSP has awarded scholarships to 

2,454 students (table 3-4). The total awards to the three priority subgroups are: 

• 508 scholarship awards to public school students attending schools designated as 
SINI the year before they entered the lottery; 

• 1,730 scholarship awards to students in non-SINI public schools; and 

• 216 scholarship awards to students attending private schools but otherwise eligible 
for the Program. 
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Table 3-4. Probability of receiving a scholarship, by applicant type and grade-level band, 
cohorts 1 and 2 combined 







Grade-level band 






Type of applicant 


K-5 


6-8 


9-12 


Total 


SINI public 


Eligibles 


301 


204 


229 


734 


Probability 


80.0% 


77.4% 


47.6% 


69.2% 


Scholarships 


241 


158 


109 


508 


Nonrecipients 


60 


46 


120 


226 


Non-SINI public 


Eligibles 


1,673 


533 


219 


2,425 


Probability 


77.9% 


68.3% 


28.3% 


71.3% 


Scholarships 


1,304 


364 


62 


1,730 


Nonrecipients 


369 


169 


157 


695 


All public 


Eligibles 


1,974 


737 


448 


3,159 


Probability 


78.3% 


70.3% 


38.2% 


71.0% 


Scholarships 


1,545 


522 


171 


2,238 


Nonrecipients 


429 


215 


277 


921 


Private 


Eligibles 


390 


278 


220 


888 


Probability 


33.9% 


23.4% 


8.6% 


24.3% 


Scholarships 


132 


65 


19 


216 


Nonrecipients 


258 


213 


201 


672 


Total (public and private) 


Eligibles 


2,364 


1,015 


668 


4,047 


Probability 


70.9% 


57.8% 


28.4% 


60.6% 


Scholarships 


1,677 


587 


190 


2,454 


Nonrecipients 


687 


428 


478 


1,593 



NOTES: The probability for applicants already attending private school was 42.8 percent for cohort 1 and zero 

percent for cohort 2. The probability for SINI public school applicants in all grade bands was 100 
percent for cohort 1; see table 3-3 for the probabilities for cohort 2. The probability for non-SINI public 
school applicants in the K-5 grade band was 100 percent for cohort 1; see table 3-3 for the probability for 
cohort 2. 

SOURCES: The DC Opportunity Scholarship Program applications and the Program operator’s files. 

3.3 Impact Sample 

The impact sample is a direct result of the lotteries and the critical component of the legislatively 
required rigorous evaluation of the OSP. Impact evaluations compare the outcomes for a group 
of study participants, some of whom were randomly awarded access to the intervention (e.g., an 
OSP scholarship), and some of whom were randomly assigned to not receive access. The 
lotteries conducted for the OSP cohorts in years 1 and 2 satisfy these requirements. Since the 
intervention under consideration is an Opportunity Scholarship to attend a private school, the 
impact analysis should focus on the population of applicants for whom private schooling 
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represented a new opportunity. Thus, the impact sample for this evaluation will comprise all 
eligible applicants who were previously attending public schools (or were rising kindergartners) 
AND were subject to a lottery to determine whether they would receive an Opportunity 
Scholarship (figure 3-2, shaded). 

Figure 3-2. Construction of the impact sample from the applicant pool, cohorts 1 and 2 




NOTES: Cl = Cohort 1 (applicants in spring 2004) 
C2 = Cohort 2 (applicants in spring 2005) 
Total = Cl and C2 



“The group of applicants who were not randomly assigned includes: in cohort 1, public school applicants from SINI 
schools or who were entering grades K-5 (all received a 100 percent probability of getting a scholarship), and in cohort 2, 
private school applicants, the lowest priority group (all received a zero percent probability of award because it was clear the 
Program would be filled with higher priority public school applicants). 



The subgroups of eligible applicants to the Program who do not fit the criteria for the impact sample include eligible 
applicants in cohorts 1 and 2 who were already attending private schools (n=888) and two groups of public school applicants 
in cohort 1 who were all automatically awarded scholarships (w=851), specifically those from SINI public schools because of 
their high service priority and those applying for grades K-5 because there were sufficient private school slots in those 
grades to accommodate all of those applicants that year. 
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Overall 



The impact sample group has the following characteristics: 

• A total of 2,308 students, with 1,387 assigned by lottery to the treatment group and 
921 assigned by lottery to the control group/^ 

• Approximately 21 percent of the impact sample is from cohort 1 (492 eligible 
applicants in grades 6-8 and 9-12), and the remaining 79 percent (all 1,816 eligible 
public school applicants) is from cohort 2. 

The more than 2,300 students is a large study sample relative to the impact samples used in other 
evaluations of private school voucher programs (803 to 1,960 students).'^ Statistical 
computations based on reasonable assumptions about study response rates indicate that the 
impact sample will be sufficient to detect Program impacts of at least a moderate and 

1 R 

educationally meaningful size. 

Treatment vs. Control Group Differences 

An important strength of experimental methods of analysis is that the assignment of study 
participants to the treatment and control groups creates two analytic groups that are statistically 
similar. The treatment, in this case the offer of an Opportunity Scholarship, is provided to one 
group, and any subsequent differences in outcomes observed between the groups can be ascribed 
to the impact of that treatment. To ensure those conditions for the impact analysis, researchers 
must compare the characteristics of the treatment and control groups prior to the Program (at 
baseline) to see if the random assignment worked. Chance alone will occasionally generate 
baseline differences between the treatment and control groups about 1 time out of 20, but a 
properly executed lottery should produce analytic groups that are similar in almost all respects. 

Analysis of the OSP groups at baseline suggests that a strong foundation for the impact analysis 
has been laid, although some statistical procedures will be used to further equate the groups. The 



A total of five members (2.5 percent) of the cohort 1 randomized control group were awarded scholarships hy lottery in the 
summer of 2005 as part of the control group follow-up lottery to reward control group members who cooperate with the 
evaluation testing requirements. Additional details regarding the follow-up control group lotteries are provided in Appendix 
A. 

William G. Howell and Paul E. Peterson, with Patrick J. Wolf and David E. Campbell, The Education Gap: Vouchers and 
Urban Schools (Washington: Brookings, 2002), 44. 

According to our power analysis in the first year report, an initial impact sample of 2,201 students should be sufficient 
to detect even rather modest test score impacts of 0.15 standard deviations hoth after 1 year and after 3 years of 
Program operation. See Patrick Wolf, Bahette Gutmann, Nada Eissa, Michael Puma, and Marsha Silverherg, Evaluation of 
the DC Opportunity Scholarship Program: First Year Report on Participation (Washington, DC: U.S. Government Printing 
Office, 2005), A-4. To place this estimated effect size in context, an effect of 0.15 equates to a Normal Curve Equivalent 
(NCE) difference of 3.15 NCE points, since one standard deviation on the SAT-9 is 21/06 NCEs. Converting NCEs to a 
change in percentile ranks depends on where on the overall distribution the observed change occurs. Eor example, if the 
control group was, on average, at the 20* percentile, a gain of 3.15 NCEs would bring it up to about the 24* percentile. 
Such a gain is likely to be considered modest but educationally meaningful, and the capability to detect even a modest 
educational change is a clear strength of the impact evaluation going forward. 
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lotteries were conducted within grade bands, so any comparisons between the treatment and 
control group need to be made within such groupings. Those comparisons for cohort 1 
determined that the first-year lotteries worked as designed in producing statistically similar 
analytic groups. However, for cohort 2, the treatment and control groups within grade-bands 
differ from each other to a statistically meaningful extent in 2 of 15 comparisons (table 3-5). For 
students entering grades K-5, the average family income of members of the control group is 
$1,287 higher than that of the treatment group, and the average years of their mother’s 
education is also slightly higher among elementary students randomly assigned to the control 
group. The treatment and control groups within the K-5 grade-band are statistically similar 
regarding race, ethnicity, and gender, and the analytic groups within the junior high and high 
school grade bands are indistinguishable on all factors measured. 

The presence of 2 statistically significant differences out of 15 comparisons between 
the treatment and control groups is most likely due to random chance. In year 2, multiple, 
small-scale lotteries were conducted for each grade band, and the odds of getting differences 
between groups with random assignment increases when the samples are small. In any case, 
because we have nearly complete baseline measures for student background factors, we can 
control for any measurable post-lottery differences between the analytic groups in the course of 
estimating subsequent Program impact. 

3.4 Scholarship Usage 

A large portion of the applicant pool and the impact sample received scholarships through the 
lotteries. These students are expected to use the funds to attend a participating private school of 
their choice. However, Opportunity Scholarship usage rates have varied by cohort, participant 
sample, and over time (table 3-6): 

• Nearly three-quarters of OSP recipients use the scholarship the first year the students 
receive them. These initial usage rates are in the mid to high-range of usage rates 
reported by previous school voucher studies.^^ 



Wolf et at, Evaluation of the DC Opportunity Scholarship Program, 41-43. 

To enable more parents to begin their school search early, within each grade band there was an early lottery for 
students deemed eligible by mid-spring and a late lottery for students not confirmed eligible until early summer. Students 
entering the same grade and with the same priority characteristics were assigned the exact same probability of winning a 
scholarship regardless of which scholarship lottery they entered. 

The initial scholarship usage rate was 82 percent in New York City, 78 percent in Dayton, Ohio, and 68 percent 
in Washington, DC, for the earlier experimental evaluation of private scholarship programs in those three cities. Voucher 
usage rates were 61 percent in the first year of the Milwaukee Parental Choice Program and estimated to be about 50 percent 
in the first year of the Cleveland Scholarship and Tutoring Program. See Howell et ah, The Education Gap, 44; John F. 
Witte, Eirst Year Report: Milwaukee Parental Choice Program (University of Wisconsin-Madison, November 1991), 3; John 
F. Witte, Andrea B. Bailey, and Christopher A. Thom, Second Year Report: Milwaukee Parental Choice Program (University 
of Wisconsin-Madison, December 1992), 8; Paul E. Peterson and Bryan C. Hassel, eds.. Learning Erom School Choice 
(Washington, DC: Brookings, 1998), 360. 
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Table 3-5. Characteristics of treatments versus controls, cohort 2 impact sample 



Grade-band/characteristics 


Treatments 


Controls 


Difference 


K-5 








Percent African American 


95.0 


92.3 


2.7 


Percent missing 


9 


7 




Percent Hispanic (any race) 


11.1 


9.3 


1.8 


Percent missing 


6 


7 




Percent Female 


50.7 


53.4 


-2.7 


Percent missing 


0 


0 




Average annual family income 


15,872 


17,159 


-1,287* 


Percent missing 


0 


0 




Average years of mother’s education 


12.6 


12.7 


-0.1** 


Percent missing 


15 


14 




Sample size 


749 


429 




6-8 








Percent African American 


93.2 


92.3 


0.9 


Percent missing 


10 


10 




Percent Hispanic (any race) 


15.8 


12.6 


3.2 


Percent missing 


8 


5 




Percent female 


48.6 


54.5 


-5.9 


Percent missing 


0 


0 




Average family income 


18,296 


17,232 


1,064 


Percent missing 


0 


0 




Average years of mother’s education 


12.5 


12.4 


0.1 


Percent missing 


19 


11 




Sample size 


247 


134 




9-12 








Percent African American 


92.8 


93.6 


-0.8 


Percent missing 


10 


5 




Percent Hispanic (any race) 


11.0 


6.7 


4.3 


Percent missing 


11 


10 




Percent female 


44.6 


46.7 


-2.1 


Percent missing 


0 


0 




Average family income 


18,577 


18,431 


146 


Percent missing 


0 


0 




Average years of mother’ s education. 


12.5 


12.7 


-0.2 


Percent missing 


21 


21 




Sample size 


92 


165 





SOURCES: The DC Opportunity Scholarship Program applications and the Program operator’s files. 
* Statistically significant at the 95 percent confidence level. 



** Statistically significant at the 99 percent confidence level. 
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• The impact sample overall has a slightly lower rate of initial use (71 
percent), which can be attributed to two factors: (1) the group excludes 
students already attending private schools at the time of application, whose 
use rates are substantially higher than those for public school recipients and 
(2) the sample includes a higher proportion of older students, a group that 
was more constrained in their choice of schools under the Program and who 
experienced substantially lower use rates. 

• Program attrition has been modest for cohort 1 students who initially used 
their scholarships. About 3 percent of the OSP recipient sample and 5 percent of the 
impact sample left the Program during the academic year after having initially used a 
scholarship to attend a private school in fall of 2004. Another 5 and 3 percent, 
respectively, left the Program during the transition to their second year. Thus, in 
cohort 1, a total of 64 percent of the OSP recipient sample and half of the impact 
sample have used their scholarships continuously since fall of 2004. 

Table 3-6. Scholarship usage rates, OSP recipient and impact samples, cohorts 1 and 2 





Cohort 1 


Cohort 2 


Totals 


Persistence 


OSP 

recipient 

sample 


Impact 

sample 


OSP 

recipient 

sample 


Impact 

sample 


OSP 

recipient 

sample 


Impact 

sample 


Number received 
scholarships 


1,366 


299 


1,088 


1,088 


2,454 


1,387 


Percent initially used 


75 


62 


73 


73 


74 


71 


Percent used all year 1 


72 


57 


NA 


NA 


NA 


NA 


Percent used year 2 


67 


54 


NA 


NA 


NA 


NA 


Percent used years 1 and 2 


64 


50 


NA 


NA 


NA 


NA 



NOTE: The OSP recipient sample consists of all students offered scholarships, regardless of the award mechanism 

(automatic or lottery) or priority status (SINI public, non-SINI public, private). NA signifies that information is not 
yet available. 

SOURCES: The DC Opportunity Scholarship Program applications and the Program operator’s files. 

As noted above, scholarship usage rates also are associated with the type of school the 
scholarship recipients previously attended and the grade they are entering. Across both 
cohorts, initial use of scholarships was lowest among students from SINI schools (69 percent) 
and, not surprisingly, highest for students who were already attending private schools when 
they applied for a scholarship (88 percent) (table 3-7). Scholarship usage rates differed even 
more based on grade band. Initial usage was highest among students entering K-5 (79 percent), 
somewhat lower among students entering 6-8 (69 percent) and lowest among students entering 9- 
12 (51 percent). 
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Table 3-7. Scholarship usage rates, by school type at time of application and grade band, 
cohorts 1 and 2 combined 



Student characteristics 


Number received 
scholarships 


Percent initially 
used 

scholarships 


School type at application 






SINI public 


508 


69 


Non-SINI public 


1,730 


74 


Private 


216 


88 


Grade hand 






K-5 


1,677 


79 


6-8 


587 


69 


9-12 


190 


51 



SOURCES: The DC Opportunity Scholarship Program applications and the Program operator’s files. 

Scholarship usage rates are important to the evaluation of a policy intervention for two reasons. 
First, usage rates send an initial signal regarding the level of enthusiasm of clients toward the 
program and their ability to navigate the steps necessary for their children to make use of the 
program. Second, one of the impact questions the evaluation is designed to address is “what is 
the impact of actually attending a private school via a scholarship program?” When program 
usage and persistence rates are relatively high, the approach for determining program efficacy 
can more clearly stem from the randomly assigned groups of students. The lower the rates of use, 
the more the evaluators have to deviate from the random assignment and the greater the potential 
for inaccurate estimates. That important issue and the other central research questions associated 
with this evaluation will be addressed with initial outcome data in the next report to Congress. 



20 





Appendix A: Congressionally Mandated Evaluation 



Section 309 of the District of Columbia School Choice Incentive Act of 2003 describes the 
requirements for an independent evaluation of the DC Opportunity Scholarship Program. The 
Secretary of Education is to ensure the following: 

• “The evaluation is conducted using the strongest possible research design for 
determining the effectiveness” of the school choice program; and 

• “The results of the evaluation regarding the impact of the program on the 
participating students and nonparticipating students and schools in the District are 
disseminated widely.” 

Early on, the Institute of Education Sciences determined that the foundation of the DC 
Opportunity Scholarship Program evaluation would be a randomized controlled trial (RCT) 
that compared outcomes of eligible applicants (students and their parents) randomly assigned to 
receive or not receive a scholarship. This decision was based on the mandate to use rigorous 
evaluation methods, the expectation that there would be more applicants than funds and private 
school spaces available, and the requirement to use random selection to determine who receives 
a scholarship. In addition, the law clearly specified that such a comparison in outcomes be 
made. This component represents the impact analysis and will provide evidence on the 
effectiveness of the Program.^'^ 

The law also called for the evaluation to track Program progress in other ways. Eor 
example, the evaluation was to compare the achievement of students participating in the 
scholarship Program to the achievement of students in the same grades in the DC Public 
Schools (DCPS). However, that performance reporting is no longer possible, since DCPS 
is in the process of changing its assessment away from the SAT-9 used when the OSP began 
and which the evaluation continues to administer. The evaluation will address other issues the 
statute requires, such as the experiences of DC schools during the period of Program 
implementation. 



RCTs are commonly referred to as the “gold standard” for evaluating educational interventions; when mere chance determines 
which eligible applicants receive access to school choice, the students who apply but are not admitted make up an ideal 
“control group” for comparison with the school choice “treatment group.” Both groups of participants are equally motivated 
to obtain new educational options, and nothing except a random draw distinguishes those who receive the opportunity from 
those who do not. Therefore, any differences in the two groups in subsequent years can be attributed to the impact of the 
Program. In contrast, the results of school choice studies that are not based on RCTs must be interpreted and used more 
cautiously because comparisons between the applicants and a group of students who chose not to apply will likely reflect not 
only the impact of the Program but also differences between the groups in motivation and other unmeasured characteristics. 

See 309(a)(4)(A)(ii). 

The RCT approach was also used by researchers conducting impact evaluations of the New York City; Dayton, Ohio; and 
Washington, DC, private scholarship programs. 
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Research Questions 



Based on guidance in the statute, the research team plans to conduct a comprehensive and 
rigorous RCT evaluation of the impact of the OSP on participating students and families. 
Specifically, the evaluation will address the following research questions: 

1. What is the impact of the Program on student academic achievement? The 

law places high priority on examining whether the Program — the availability 
and offer of scholarships — improves the academic achievement of eligible 
students. This question can be addressed most rigorously by comparing the 
academic achievement of student applicants randomly assigned by a lottery 
to receive and not receive scholarships. 

2. What is the impact of attending private versus public schools? Because 
some students offered scholarships will choose not to use them, the research 
team will use accepted statistical methods to examine the effects for students 
who take the scholarship offer and successfully enroll in a private school. 

3. What is the impact of the Program on other student measures? The law 

calls for examining other indicators of student and school success, including 
persistence, retention, graduation, and, if possible, college enrollment. In 
addition, the legislation requires the evaluation to assess the school safety 
of students who receive the scholarships relative to those who did not receive 
scholarships. 

4. What effect does the Program have on student and parent satisfaction 
with the educational options available in the District and with children’s 
actual school experiences? A key desired outcome of scholarship programs is 
an increase in both the school choices possible and parents’ and students’ 
satisfaction with the choices they have made. These issues will be examined by 
comparing the satisfaction and reasons for applying to the DC Opportunity 
Scholarship Program among applicants assigned by lottery to receive 
scholarships and those assigned to not receive scholarships. 

5. To what extent is the Program influencing public schools and expanding 
choice options for parents in Washington, DC? Scholarship programs have 
been hypothesized to affect not only the students who receive the 
scholarships but also the broader population of public schools and 
students. Theory suggests that these broader outcomes could occur if a 
significant number of students move from public to private schools. The 
public schools might experience a reduction in per-student funding that affects 
their offerings, a change in average student performance, or they may respond 
to the competition for students by changing curricula, adopting new 
themes or missions, and modifying existing policies and practices to 
make the public schools more attractive to students with schooling 
options. Choice programs might also affect the larger population of 
private schools, beyond those in which the programs’ participants are 
currently enrolled; if choice programs are successful, additional private 
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schools may choose to participate, new schools may be established to meet 
enrollment demand, or existing schools might expand capacity. These issues 
will be explored with descriptive analyses based primarily on school 
surveys. 



Data Collection 

To answer these questions, the evaluation will draw on different types of data — some available 
from DCPS, some collected for the purposes of this study. These data will include pre-Program 
(baseline) measures of family background and student achievement. The baseline measures allow 
us to verify that students randomly assigned to the scholarship and nonscholarship groups were, 
in fact, similar before the Program; the measures also enable us to create subgroups of students 
whose impacts we might want to examine separately, such as students with low prior 
achievement. Additional data collected will include annual “in-Program” measures (e.g., parents’ 
satisfaction with their children’s school, students’ academic achievement), which will serve as 
outcomes for the rigorous evaluation of Program impacts (table A-1). 

Impact Analysis 

It is well-known that the independent effects of school choice on student outcomes are 
difficult to estimate. Perhaps the most significant difficulty faced by researchers is selection 
bias — the self-selection of families to even seek out a new school choice for their child, and the 
mutual student/school decision process that selects students into different types of schools. 
Because this bias is generally a result of unmeasurable factors, most researchers have 
preferred the use of an RCT to a dependence on nonexperimental (nonrandomized) 
statistical methods. Since the DC Opportunity Scholarship Program provides for the 
random distribution of scholarships through a lottery, we will use RCT methods to estimate 
most Program impacts. 

Impact Analysis Sample 

The RCT approach rests on random assignment or, in the case of the DC Opportunity 
Scholarship Program evaluation, a lottery to create two statistically equivalent groups of 
students from among Program applicants: a “treatment” group that receives a scholarship, and a 
“control” group that does not receive a scholarship. Because the two groups are generated from 
the same pool of applicants, they are equally likely to be motivated to participate in the Program 
and to reap any benefits from it. And as long as the pool of applicants is sufficiently large, 
the random assignment of students into treatment and control groups should produce groups 
that are similar in other characteristics, both those we can observe and measure (e.g., family 
income, prior academic achievement) and those we cannot (e.g., motivation to succeed). The 
random assignment ensures that all observed and unobserved characteristics are equally 
represented in both groups. 
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Table A-1. Data sources 



Data source"' 


Description 


Student assessments 


• Baseline, or pre-Program measures of student achievement for 
public school applicants will come primarily from the SAT-9 
standardized assessment administered by DCPS as part of its 
spring testing program.'’ 

• Each spring after the baseline year, the study will administer the 
SAT-9 to all students who were offered scholarships as well as all 
members of the control group who did not receive a scholarship. 

. DCPS test score data will be obtained for all public school students 
in those years, to compare with the scores of students participating 
in the Program." 


Parent surveys 


. Surveys of parents (of all applicants) will be conducted in all 4 
years of impact evaluation data collection. 

. Topics will include reasons for applying, satisfaction with school 
choices, perceptions of school safety, educational climate, and 
offerings. 


Student surveys 


. Each year after baseline, surveys will be conducted with all 
applicants who are in grades 4 and higher. 

• Topics will include students’ satisfaction with their schools, 
perceptions of safety, and other characteristics of their school 
program and environment. 


Principal surveys 


• Surveys will be conducted each year of principals of all public and 
private schools then operating in DC, including public charter 
schools. 

• Topics will include school organization, safety, climate, principals’ 
awareness of and response to the DC Opportunity Scholarship 
Program, and, for private school principals, why they are/are not 
participating. 



The research team originally planned to administer surveys to teachers, but this plan was dropped due to the higher than expected 
cost of student testing and the resistance of private school administrators to the burden of teacher-level data collection. 

’’ Baseline achievement will be collected only for applicants from public schools; applicants who were already attending 
private schools will not be included in the impact analysis. For public school applicants who did not participate in regular DCPS 
testing in the year they applied to the Program (e.g., particularly children below grade 3), the study will administer the 
equivalent DCPS assessment to these students as soon as possible after application. All other data will be collected for all 
applicants, both from public and private schools. 

‘ Beginning in spring of 2006, DCPS will administer a customized, criterion-referenced achievement test (CRT) for its 
accountability testing instead of the SAT-9. Since scores on the CRT are not directly comparable to scores on the SAT-9 
administered for purposes of the evaluation, the performance report, which relies on DCPS accountability testing data for 
comparison, will be less reliable and informative than if the District were continuing to administer the SAT-9. The switch from 
the SAT-9 to the CRT will not affect the impact analysis, since members of both the treatment and control groups will be 
administered the same test (the SAT-9) by the evaluation team for purposes of determining Program impact. 
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According to the statute, the random assignment that is the means to create the treatment 
and control groups can only be used to help allocate scholarships under particular 
circumstances. As a result of these conditions, the impact analysis sample will: 



• Exclude applicants already enrolled in private school when they applied to the 
OSP. The statute contained no provision to exclude from the Program students who 
were currently enrolled in private schools but otherwise eligible to participate. A 
substantial number of private school students did apply to the Program, especially 
in the first year of Program implementation. However, because those students 
intended to use the DC Opportunity Scholarship to continue to attend private 
schools, measuring the difference in outcomes between private school applicants 
who did and did not receive a scholarship through the lottery would likely only 
answer the question of whether a different type or amount of scholarship funds 
affects student outcomes. While that question is of some policy interest, it is not the 
main focus of the evaluation as specified in the legislation. Therefore, 
applicants already enrolled in private schools when they applied are not part of the 
impact analysis sample. 

• Exclude any students who were originally assigned to the control group by 
lottery but subsequently were awarded scholarships by way of follow-up 
lotteries. The participants in the impact evaluation all sought scholarships to attend 
private schools. To encourage those randomly assigned to the control group 
(nonrecipients) to turn out for follow-up data collection, a special lottery is held each 
spring in which the only eligible students are those originally assigned to the control 
group who subsequently cooperated with outcome data collection. Because they 
represent study-induced treatment crossover (i.e., participants assigned to the control 
condition who are subsequently offered the treatment) and were chosen at random, 
these students will be excluded from the subsequent impact analysis. Five 
scholarships were awarded to control group students by lottery in the summer of 

2005, when the control group totaled 193 students. Ten scholarships will be awarded 
to control group students by lottery in the spring of 2006, when the control group will 
number 911 (e.g., 921 minus 5 control awardees minus 5 high school graduates). Ten 
or more new scholarships will be awarded to control group students each year after 

2006. The number of scholarships awarded to members of the control group needs to 
be kept modest in the initial years of an experimental evaluation so that the control 
group remains sufficiently large to enable researchers to identify Program impact. 

• Include only public school applicants in grades where there are more 
applications than there are available private school slots. A lottery is a fair and 
efficient way of distributing scholarships when there are too many applicants, but is 
inappropriate as an allocation device when sufficient scholarship funds and private 
school slots exist to accommodate all the eligible applicants at certain grade levels. 
In those grade levels, all applicants will receive scholarships, and they will be 
excluded from the impact sample. 



Some of these applicants from private schools were already relying on scholarship funds in order to attend those 
schools. However, the scholarships they were receiving may have heen less generous than those available under the DC 
Opportunity Scholarship Program. 
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Thus, the impact evaluation of the DC Opportunity Scholarship Program depends on the extent 
to which large numbers of eligible DC families with public school students apply to the Program. 
The treatment and control groups must be of a sufficient size to allow us to detect and 
measure any difference in outcomes between the two groups (the “impact”) with statistical 
certainty. A procedure called “power analysis” is used to determine the sample sizes necessary to 
enable the study to answer the central research questions and to measure Program effects that are 
large enough to be both meaningful in students’ lives and relevant to policy debates about the 
efficacy of school choice interventions. At the end of the 18-month initial implementation 
period, we know the following about the impact sample and study power: 

• Cohort 1 includes 492 eligible applicants who qualify as members of the impact 
sample. They all were entering grades 6-12 in fall of 2004. A total of 299 are members 
of the treatment group, and 193 are members of the control group. 

• Cohort 2 includes 1,816 eligible applicants who qualify as members of the impact 
sample. They cover all eligible grades, K-12. A total of 1,088 are members of the 
treatment group, and 728 are members of the control group. 

• The combined-cohort impact sample totals 2,308 students, of whom 1,387 are 
members of the treatment group, and 921 are members of the control group. 

• Preliminary estimates of study power, given these participant numbers, suggest that the 
analysis will be able to detect even modest but educationally meaningful Program 
impacts for the combined-cohort sample and for the Cohort 2 sample alone. 

General Statistical Approach: Estimating the Impact of the Offer of a Scholarship 

Given appropriately sized treatment and control groups, the strategy for analyzing 
impacts is well established. To motivate the discussion of how we identify the effect of the 
scholarship Program on test scores, it is useful to begin with a simple representation of the 
selection problem as a missing data problem, using the potential outcomes approach. This 
approach defines causal effects in terms of potential outcomes or counterf actuals. 
Conceptually, the causal effect of treatment — the scholarship — is defined as the difference 
between the “outcome for individuals assigned to the treatment group” and the “outcome for the 
treatment group if it had not received the treatment,” or: 

(E.l) “E(Yi I Xi, Ti =1)” - “E(Yi I Xi, T; =0)” 

In the case of scholarships, the treatment effect — the effect of the scholarships on 
academic achievement — would be defined as the difference between “test scores for Program 
students” and “test scores for Program students if they had not received a scholarship.” The 
fundamental problem is that a student is never observed simultaneously in both states of the 
world. What is observed is a student in the treatment group (Ti =1) or in the control group 
(Ti =0). The outcome in the absence of treatment, E(YilXi, Ti =0), is then the counterfactual — 
what would have occurred to those students receiving the scholarships if they had not received 
them. 
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If students receiving scholarships were identical to other students in both observable and 
unobservable characteristics, the counterfactual could be generated directly from an 
appropriately selected comparison group. Valid comparison groups are rarely found in 
practice, however. The random assignment of students into the Program generates the 
counterfactual from the control group — eligible applicants who did not receive a scholarship.^^ 
If correctly implemented, random assignment yields statistically equivalent groups and allows 
estimation of the Program impact through differences in mean outcomes between the two 
groups. 

Consistent with this approach is the following basic analytic model of the effects of school 
choice scholarships on outcomes. Consider first the outcome equation for the test score of 
student i in year t. It is reasonable to assume that test scores (Yit) are determined as follows: 

(E.2) Yit = a + X Tit + Xi Y + s it if t>k (period after Program takes effect) 

Equation (E.2) estimates the effect of the ojfer of a scholarship on student outcomes. Under this 
model, commonly referred to as the “Intent to Treat” (ITT) estimation, all students who were 
randomly assigned by virtue of the lottery are included in the analysis, regardless of whether 
a member of the treatment group uses the scholarship to attend a private school. In E.2, Tit is 
equal to one if the student has the opportunity to participate in the scholarship Program (i.e., the 
award rather than the actual use of the scholarship) and equal to zero otherwise. Xi is a vector of 
student characteristics (measured at baseline) known to influence future academic achievement, 
such as prior test scores, mother’s level of education, family income, etc. In this model, x 
represents the effect of scholarships on test scores for students in the Program, conditional on Xi. 
With a properly designed RCT, using a judiciously chosen set of statistical controls for 
characteristics that predict future achievement should improve the precision of the estimated 
impact. That treatment effect, x, should be identical to the difference in mean outcomes 
between the treatment and the control groups. 

Since the initial applicants were randomized within certain relevant subgroups, we will analyze 
Program impacts using a randomized block design. We are interested in how academic 



See the following studies, which all use the same data from an evaluation of a New York City privately funded scholarship 
Program: William G. Howell, Patrick J. Wolf, David E. Campbell, and Paul E. Peterson, “School Vouchers and Academic 
Performance: Results from Three Randomized Field Trials,” Journal of Policy Analysis and Management 21 (2000): 2; John 
Barnard, Constantine E. Frangakis, Jennifer L. Hill, and Donald B. Rubin, “Principal Stratification Approach to Broken 
Randomized Experiments: A Case Study of School Choice Vouchers in New York City,” Journal of the American 
Statistical Association 98 (2003): 462; Alan B. Krueger and Pei Zhu, Another Look at the New York City School Voucher 
Experiment, Working Paper Series, Education Research (Princeton, NJ: Princeton University, March 2003). 

For a spirited debate about the use of this technique in the context of school choice research, see William G. Howell and Paul 
E. Peterson, “Uses of Theory in Randomized Field Trials: Lessons from School Voucher Research on Disaggregation, 
Missing Data, and the Generalization of Findings,” American Behavioral Scientist 47 (Jan. 2004): 634-657; Krueger and 
Zhu, Another Look, 658-698; Paul E. Peterson and William G. Howell, “Efficiency, Bias, and Classification Schemes: A 
Response to Krueger, A.B. and Zhu, P., ‘Another Look at the New York City School Voucher Experiment,”’ Working Paper 
Series, Education Research (Princeton, NJ: Princeton University, March 2003): 699-717; Alan B. Kruger and Pei Zhu, 
“Inefficiency, Subsample Selection Bias, and Nonrobustness: A Response to Peterson, P.E. and Howell, W.G., ‘Another 
Look at the New York City School Voucher Experiment,”’ ^ Working Paper Series, Education Research (Princeton, 
NJ: Princeton University, March 2003): 718-728; Paul E. Peterson and William G. Howell, “Voucher Research 
Controversy: New Looks at the New York City Evaluation,” Education Next 4 (Spring 2004): 73-78. 
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achievement (Y) is affected by the assignment into the scholarship Program within each block 
(B) or group of size n. The impacts are then estimated as: 



(E.3) Yikt = p + X Tikt S'’j=2Pj Bik + Xik Y+ Sik,t 

where 

i = 1, ,n observations and k=l,....,b blocks (defined by grade and priority status); 

Yjiis the outcome for student i in block), at time t; 
p is the overall mean outcome (e.g., test score); 

X is the treatment (scholarship Program) effect; 

Pj is the j* block effect; 

Tit is assignment into the scholarship Program; 

Bji is the block assignment; 

Xji represents observable characteristics, measured at baseline; and 
Sij is the random error; independent, N(0,ag^ ). 

This analytical framework follows naturally from the group randomization and is easily 
implemented and interpreted. Y can be measured in several different dimensions, including test 
scores, school satisfaction, parental satisfaction, grade completion, including where appropriate, 

high school graduation, etc. p is average outcome for all Program members, p; is the 

28 

average block effect, and x is the effect of scholarships on academic achievement. 

Estimating the Impact of the Use of Scholarships 

Even with a properly implemented RCT, we may expect that not all applicants placed by 
random assignment into the treatment (scholarship offer) group will actually use the scholarship 
at a private school. That is, some scholarship recipients may choose not to use their scholarship 
and instead attend a public school. This type of nonparticipation or underutilization of treatment 
services has been observed across all RCT settings, including medical trials, job training and 
health insurance experiments, as well as in previous school scholarship RCTs such as the one of 
the Milwaukee Parental Choice Program. 

Policymakers are typically interested in the effect of scholarship use on student achievement, in 
addition to the offer of the scholarship. To estimate the impact, we will use a model commonly 
referred to as the “Impact of the Treated” (lOT), which statistically estimates the impact of actual 
scholarship use, relying on what is called the “Bloom Adjustment.” This is possible by using 
the original comparison of all treatment group members to all control group members but 
interpreting it in a different way. The new interpretation says that the treatment group’s impact — 
how its outcomes differ from what would have transpired without a scholarship — has two 
components: 



Depending on the extent to which the randomly assigned applicants are clustered in their schools, some adjustments to the 
standard error estimates may be necessary. 

Howard S. Bloom, “Accounting for No-Shows in Experimental Evaluation Designs.” Evaluation Review, 8 (1984): 225-246. 
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• The impact on the decliners, who by definition do not participate in the Program even 
though offered a scholarship, which can logically be assumed to be zero. 

• The impact on everyone else assigned to the treatment group — i.e., on the OSP 
participants who make up the rest of the experimentally determined treatment group. 

This assumption alone — the presumption that the decliners remain unaffected by their 
assignment to the treatment group — makes it possible to translate the measured effect of the 
scholarship Program on the entire treatment group (which the experimental design provides 
directly as described above) as a way to assess the average effect of the Program on just the 
participants. It does not matter what the average effect would have been on the decliners had 
they participated. Nor does it matter whether decliners have different outcomes than 
participants due to “selection” or pre-existing differences. Thus, we will use a simple Bloom 
Adjustment to estimate the impact of the OSP on actual scholarship users. 

Reports 

The law requires the Secretary to submit to the Congress annual reports resulting from the 
independent evaluation and a final report not more than a year after the 5-year Program ends. 
These reports should provide the Congress, other policymakers, the research community, and the 
public at large with important new information about what happens to students, families, schools, 
and communities when educational options are expanded for urban low-income families through 
public policy. The next report, the first with results from the impact analysis, will be submitted in 
spring 2007. 



A-9 





www.ed.gov 



ies.ed.gov 





